Goto

Collaborating Authors

 Elk Grove


Spurious Forgetting in Continual Learning of Language Models

Zheng, Junhao, Cai, Xidi, Qiu, Shengjie, Ma, Qianli

arXiv.org Artificial Intelligence

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.


For people who stutter, the convenience of voice assistant technology remains out of reach

USATODAY - Tech Top Stories

Do you ever feel as if your voice assistants – whether Siri, Alexa, or Google – don't understand you? You might repeat your question a little slower, a little louder, but eventually you'll get the information you were asking for read back to you in the pleasing but lifeless tones of your voice-activated assistant. That's the question facing many of the 3 million people in the United States who stutter, plus the thousands of others who have impaired speech not limited to stuttering, and many are feeling left out. "When this stuff first started coming out, I was all over it," said Jacquelyn Joyce Revere, a screenwriter from Los Angeles who stutters. "In LA, I need GPS all the time, so this seemed like a more convenient way to live the life I want to live."


Inventive snapper creates stunning moon photos by combining 50,000 photos

Daily Mail - Science & tech

A cameraman has taken backyard photography of the night sky to the next level by creating a stunningly clear image of the moon that combines 50,000 separate snaps. Andrew McCarthy, from Elk Grove, California, took the stunning 81 megapixel shot that shows half the moon shrouded in darkness while the other is illuminated. Rather than using high-tech equipment costing tens of thousands, Andrew simply used two regular cameras to capture the different parts of the moon, then stitch them together using computer software. The final image is a whopping 81 megapixels in size - around three times higher resolution than found in cutting edge 8K TV sets - because of this unique approach. Two regular cameras caught different parts of the moon. Sections of the moon were captured in what Mr McCarthy calls'tiles' - images stitched together using Photoshop - before the best and clearest shots were chosen, averaging out factors like blurring.


Pokémon GO new game sees flirty propositions being sent between strangers

Daily Mail - Science & tech

Gaming is often seen as an unsociable and solitary pursuit. But fans of Pokémon GO claim the augmented reality craze - which since its launch last week has had users chasing around their neighborhoods in pursuit of fictional creatures - has given rise to so many chance meetings between strangers that people have started using it as a dating app. Numerous users have shared their stories of romance spurred by the app - claiming that they were either asked out while playing the game or invited on a Pokémon GO date with their potential love interest. It has also proved a hit as a dating activity among existing couples and as a new way of making friends. Such is the popularity of the game - which has already been installed on more Android devices than dating app Tinder - that some fans are using it to replace movie nights, giving rise to the phrase'Pokémon GO and chill', adapted from'Netflix and chill'.